Heretofore the ability to integrate real-time three-dimensional hand interaction into software applications has been somewhat limited and has required a relatively high level of skill among practitioners. Complex three-dimensional worlds have not been easy to simulate and frequently such simulations were performed in a rudimentary manner as the required results would not support the massive efforts needed to understand the simulated to environment, objects in that environment, or the input/output devices with interaction with the environment and objects would be made. It also has it not been practical to develop complex scene graphs and mapping between the several scene graphs that might typically be present. System interaction and the visual and tactile/force feedback were also in need of improvement, particularly where it was desired that visual feedback cues be synchronized with tactile and force cues, especially for hand grasping interactions of virtual objects.
These and other limitations have been addressed by the inventive system, method, data structures, computer program, and computer program product of the invention.